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METHOD FOR SOLVING STOCHASTIC CONTROL PROBLEMS OF LINEAR 

SYSTEMS IN HIGH DIMENSION 



BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

The present invention relates generally to a method for solving stochastic control 
problems of linear systems in high dimensions. 

2. Description of Related Art 

(Note: This application references a number of different publications as indicated 
throughout the specification by reference numbers enclosed in brackets, e.g., [x], A list 
of these different publications ordered according to these reference numbers can be found 
in the Section entitled ''References" in the "Detailed Description of the Preferred 
Embodiment." Each of these publications is incorporated by reference herein,) 

Computer-implemented Supply Chain Management (SCM) applications are 
designed to link a cohesive production and distribution network and thus allow an 
enterprise to track and streamline the flow of materials and data through the process of 
manufacturing and distribution to customers, SCM applications represent a significant 
evolution from previous enterprise resource planning (ERP) systems 

One goal of SCM applications is to decrease inventory costs by matching 
production to demand. SCM applications utilize extremely complex forecasting and 
planning algorithms to predict demand based upon information stored in the enterprise's 
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database. These applications also incorporate any changes in supply chain data into the 
forecast much faster than previous modes of calculation, allowing enterprises to more 
accurately predict demand patterns and schedule production accordingly. 

Another goal of SCM applications is to reduce overall production costs by 
streamlining the flow of goods through the production process and improving 
information flow between the enterprise, its suppliers, and its distributors. Logistics- 
oriented systems, such as transportation, warehouse management, and factory scheduling 
applications, all contribute to reduced production costs. By ensuring real-time 
connectivity between the various parties in a supply chain, these applications decrease 
idle time, reduce the need to store inventory, and prevent bottlenecks in the production 
process. 

Yet another goal of SCM applications is to improve customer satisfaction by 
offering increased speed and adaptability. SCM applications allow the enterprise to 
reduce lead times, increase quality, and offer greater customization, enhancing the 
customer relationship and improving retention. 

SCM applications begin with forecasting and data mining applications analyzing 
information consolidated in the enterprise's database. Planning algorithms are used to 
generate a demand forecast upon which to base subsequent procurement orders and 
production schedules. 

Nonetheless, there is a need in the art for improved planning techniques for SCM 
applications, especially where a SCM application models a Markov Decision Process 
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(MDP), and the action space and the state space of the MDP model are continuous and 
related to each other through a system of linear constraints. 

SUMMARY OF THE INVENTION 
To overcome the limitations in the prior art described above, and to overcome 
other limitations that will become apparent upon reading and understanding the present 
specification, the present invention discloses a method, apparatus, and article of 
manufacture for solving stochastic control problems of linear systems in high dimensions 
by modeling a structured Markov Decision Process (MDP). A state space for the MDP is 
a polyhedron in a Euclidean space and one or more actions that are feasible in a state of 
the state space are linearly constrained with respect to the state. One or more 
approximations are built from above and from below to a value function for the state 
using representations that facilitate the computation of approximately optimal actions at 
any given state by linear programming. 

Various advantages and featiu-es of novelty, which characterize the invention, are 
pointed out with particularity in the claims annexed hereto and form a part hereof 
However, for a better understanding of the invention, its advantages, and the objects 
obtained by its use, reference should be made to the drawings which form a frirther part 
hereof, and to accompanying descriptive matter, in which there is illustrated and 
described specific examples of an apparatus in accordance with the invention. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



Referring now to the drawings in which Uke reference numbers represent 
corresponding parts throughout: 

FIG. 1 is a block diagram illustrating an exemplary hardware environment used to 
implement the preferred embodiment of the present invention; and 

FIG. 2 is a flowchart that illustrates the general logic of a supply chain plarming 
process according to the preferred embodiment of the present invention. 



In the following description of the preferred embodiment, reference is made to the 
accompanying drawings, which form a part hereof, and in which is shown by way of 
illustration a specific embodiment in which the invention may be practiced. It is to be 
understood that other embodiments may be utilized and structural changes may be made 
without departing from the scope of the present invention. 



The present invention relates to structured Markov Decision Processes (MDP) 
where the state space is a polyhedron in a Euclidean space and the actions that are 
feasible in a state are linearly constrained with respect to the state. The present invention 
builds approximations from above and from below to the value function, using 
representations that facilitate the computation of approximately optimal actions at any 
given state by linear programming. 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 



Overview 
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Environment 

FIG. 1 is a block diagram illustrating an exemplary environment used to 
implement the preferred embodiment of the present invention. One or more client 
5 computers 100, suppher systems 102, production systems 104, and/or distribution 

systems 106 communicate with a server computer 108. Each of the cHent computers 100, 
supplier systems 102, production systems 104, distribution systems 106, and the server 
computer 108 are typically comprised of one or more processors, memory, and other 
Q components, such data storage devices and data communications devices. 

ijl 10 The client computers 100, supplier systems 102, production systems 104, and/or 

O 

"2 distribution systems 106 typically execute one or more computer programs operating 

i j| 

i_5. 

! ^ under the control of an operating system. These computer programs transmit requests to 

•= the server computer 108 for performing various functions and receive data from the 

ry server computer 108 in response to the requests. 

□ 15 The server computer 108 also operates under the control of an operating system, 

and executes one or more computer programs, such as an interface 110, supply chain 
planning process 1 12, and database management system 1 14. The interface 1 10, supply 
chain planning process 112, and database management system 114 perform various 
functions related to supply chain management (or other applications), and may transmit 
20 data to the client computers 100, supplier systems 102, production systems 104, and/or 
distribution systems 106. 
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The server computer 108 manages one or more databases 116 stored on one or 
more data storage devices. In a preferred embodiment, the databases 116 store one or 
more vectors used by the supply chain planning process 112, such as resource vectors, 
cost vectors, action vectors, and other vectors. These vectors may be generated, inter 
alia, by an enterprise resource planning (ERP) system, a point-of-sale (POS) system, or a 
manufacturing supply and distribution (MSD) system. Those skilled in the art will 
recognize, however, that other embodiments may use different databases, or different 
programs to access the databases. 

Generally, the interface 110, supply chain planning process 112, database 
management system 1 14, and database 116 each comprise logic and/or data that is 
tangibly embodied in or retrievable from a computer-readable device, medium, carrier, or 
signal, e.g., a data storage device, a remote device accessible via a data communications 
device, etc. Moreover, these logic and/or data, when read, executed, and/or interpreted 
by the server computer 108, cause the server computer 108 to perform the steps necessary 
to implement and/or use the present invention. 

Thus, the present invention may be implemented as a method, apparatus, or article 
of manufacture using standard programming and/or engineering techniques to produce 
software, firmware, hardware, or any combination thereof. The term "article of 
manufacture" (or alternatively, "computer program product") as used herein is intended 
to encompass a computer program accessible from any computer-readable device, carrier, 
or media. Of course, those skilled in the art will recognize many modifications may be 
made to this configuration without departing from the scope of the present invention. 
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Those skilled in the art will recognize that any combination of the above 
components, or any number of different components, including computer programs, 
peripherals, and other devices, may be used to implement the present invention, so long 
as similar functions are performed thereby. 



Supply Chain Planning Process 

hi the preferred embodiment, the supply chain planning process 112 preferably 
comprises a Markov Decision Process (MDP). MDPs were introduced by Bellman [1], 
and can be abstractly described as follows. The state of a system changes, alternately, by 
ifi 10 random transitions and by chosen actions. Before applying a chosen action, if the system 

y is in state s, then an action jc e X(s) (wherein X(s) is the set of possible actions at s) 

\ n 

■a- s 

can be taken at a cost of c(jc) , and then the state of the system changes to s' = g(s, x) . A 
Q poHcy is an assignment of an action jc = x(s) to each state s. If, before a random 

transition, the system is in state then the subsequent state is a random variable S whose 
1 5 probability distribution depends only on s: Pr(5 = 5' | s) = p{s, s* ) . Given an initial state 

5, a policy induces a sequence of random variable costs Cj , . . . corresponding to the 
(a priori random) actions X\X^ , . . . mandated by the policy, i.e., C, = c(X') . The 

Zoo ' 
.^^ A C,. where 0 < A, < 1 is a given 

constant, called a discount factor. An optimal policy is one that minimizes the expected 
20 total discounted cost E[C] for any given initial state. The value of a state 5 is the 
minimum of E[C] , starting at s (and applying a random transition first) over all the 
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possible policies. The value function L{s) assigns to each state s the minimum possible 
expected total discounted cost, where the initial state is s, and the system first undergoes 
a random transition, followed by transition caused by a chosen action. 

The size of the state space is a major obstacle for the practicality of MDP. If the 
5 optimal policy has to be computed explicitly in advance, then obviously, the number of 
states should be quite limited. Usually, it suffices to compute the value function, but still, 
if all the possible states have to be handled expHcitly, then the number of states must be 
limited to several millions. This number is quite small when the state space is generated 
by some state variables. In particular, if the state variables are continuous, then any 

10 reasonable discretization of them would give rise to an enormous mmiber of states that 
would prohibit solution by the standard methods of discrete MDP (see Puterman [4]). 

Other approaches to the problem are surveyed in Gordon [3], The model 
discussed in Gordon [3] and the solution approach rely heavily on linear programming. 
They, however, should not be confused with other applications of linear programming 

1 5 methods for solving MDP, either exactly or approximately, for example (see. Trick and 
Zin [5]). 

The present invention considers a model where the action space and the state 
space are continuous and related to each other through a system of linear constraints. 
This is the case in real life systems of supply chain management. It can be shown that the 
20 value function is convex and this important characteristic can be exploited for efficiently 
"learning" the value function in advance and representing it in a way that allows for real- 
time choice of actions based on it. The function is approximated both fi-om above and 
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from below by piecewise linear and convex functions. The domains of linearity of these 
functions are not stored explicitly, since such a representation would prohibit solutions in 
high dimension. Yet, hnear programming formulation allows optimizing and updating 
these functions in real-time. 

5 

The Underlying Process 

The specific process considered herein is described as follows. States are 
described by real m -vectors, and actions are described by real n -vectors. At a state s , 
the system first undergoes a transition to a state s-\-b where ^? is a random vector from a 

\ji 10 certain probability distribution over 9?"^. Furthermore, 6 is (stochastically) independent 
of 5. The action x e SR" and the state s'e 91" of the system after x is taken must satisfy 

Li the following constraints: 

1=1 s'= s + b- Ax 

;T1 

■ E- ■ 

Ms' >a md Dx>d 

1 5 where .4 € 9^"" " , Z) e 9?"^, and M, D, a and d are arrays of consistent dimensions. Thus, 

s* is determined by x and s. Taking an action x costs c^x (where the superscript T 
stands for matrix transposition). The cost vector c may itself be drawn from a 
probability distribution. 

Consider the following example. One application of the model is in terms of a 
20 dynamic production problem. Suppose a manufacturer optimizes profit by minimizing 
c^x subject to Ax<q. This is the classical (one stage) product mix problem. This 
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problem generalizes as follows. Suppose, after each production stage, the vector of 
available resources q is updated. More precisely, suppose, at the end of a production 
stage, the leftover amounts of resources are given in a vector s. Next, additional 
amounts, given in a vector b , become available, so the resources available for the next 
5 stage are given in the vector s + b , If the new production plan is given by x , then the 
leftover amounts after the next stage are given by 5' , where Ax-^- s'= s -^b . hi addition 
to that, X and s' may have to satisfy some Unear constraints. A myopic approach to the 
problem would choose x so as to minimize c^x subject to Ax-^ s'^ s + b,Ms'>a and 
Dx>d, ignoring the value of leftover resources for the fiiture. If the value ftmction !(•) 
10 is known, then an optimal policy would minimize in each stage the objective ftmction 
c^x + lL{s') subject to the above constraints. 

For any state 5, let F(s) denote the set of all the pairs (x, s' ) such that Ax+ s' =s, 
Ms' > a and Dx > d. The Bellman [1] optimality equation characterizes the value 
function: 

15 L{s) = EJm\n{c^x + XL{s') \ (jc,5') e F(s + b)}] 

where Etc denotes the expectation operator with respect to the distribution of the change 
vector b and the cost vector c. 

Convexity of the Value Function 
20 In this section, it will be proved that the ftmction !(•) is convex. 
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Let any sequence b^, b^,b^,,., of realizations of the state change vectors 
(corresponding to the time stages) be fixed, let any sequence c\ c^,. . . of realizations 
of the cost vectors (corresponding to the time stages) be fixed, and let an optimal policy 
be fixed. Given an initial state / and assuming the first change is by a random transition, 
the future (i.e., the states and actions) is now completely determined and the total 
discounted cost is equal to: 

(=0 

where x\ x^,,.. are the action vectors, necessarily satisfying Dx'> d. Furthermore, a 
sequence of state vectors s^, s\ s^,... is also determined, which satisfies 
Ax' + s''^^ ==s' + H and Ms'>a . Another initial state P would generate different 
sequences of action vectors /, y\y^,. . .and state vectors t\ so that: 

V{t'-Ac) = f^X{c'yy 

(=0 

and Ay' + + b\ Mi > a and Dy' > d. Given any 0 < a < 1 , consider the initial state 

(1-a) / +a The sequence of action vectors z' = (1-a) x' + ay' and state vectors 
w' = (1-a) y + a / also satisfy the constraints: Az' +m'"^^ = w' + Z?', Mu' >a and Dz' > d, 
and the resulting total discounted cost is equal to: 

f^Mic^Yz^ =(l~a)V{s';b,c) + aV(t';b,c) 

1=0 

By taking expectations over b and c on both sides of the latter. 
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V(cy z'] < (l-a) L(s^} + aL{P) is obtained. The value on the left-hand side 

corresponds to a policy that may not be stationary, but since there exists an optimal 
stationary poHcy, it follows that its value cannot be smaller than L{u^). Thus, 
L{u^) < (1-a) L{s^) + aL(t^). This completes the proof of convexity. 

hi view of the convexity o{L(s), it can be efficiently approximated both from 
above and from below. Both approximations can be repeatedly refined both during pre- 
processing and online. 

Approximating the Value Function from Above 

The approximation from above is based on knowledge of upper bounds on L{s) at 
each member of some manageable set of states. Suppose it has been concluded that for 
certain state vectors m*,..., m*, L(u^) < fi{i= l,.-? Then, convexity implies that for any 

nonnegative>' ={yx,...,ykY swch that [^.j/,. = 1, necessarily, L >''"') - Z/=i>'// * 
Thus, for any given state 5, the least upper bound on L{s) that can be derived from 
/= (/iv>/t)^by convexity, which is denoted by Lf{s), can be obtained by solving a 
linear programming problem as follows. 

Denote t/- w*] € i?*""'* and e - (l,...,lf € 9i\ Then, solve: 

Minimize fy 

y 

subject to Uy = s (PI) 
e>= 1 
y > Q 



12 



ARC-00-0030-US1 





A nice feature of this approximation is that there is no need to derive such least 
upper bounds ahead of time. The problem (PI) can be embedded in a larger linear 
programming problem that determines an approximately optimal action. Specifically, if 
the upper bounds are sufficiently close to L, then an approximately optimal action at a 
state s + b can be calculated by solving the following: 



The problem {P2) can be solved for any choice of the vectors b and c. If the latter are 
sampled sufficiently many times from the given probability distribution, then an updated 
approximate value function can be obtained for L{s) as follows. Denote by H(s;b,c) the 
optimal value of (PI), and denote by J{s\b,c) the minimum of c^x + k Lis") subject to 
(;c, s') G F{s+b). Obviously, J{s\b,c) <H{s;b.c), so L(s) = Ebc [J(s\b,c)] < Etc [H{s\b,c)]. 
If {b\c\..„b'',c'} is an adequate random sample, consider the values H(s;b',c'), and based 
on them calculate a high-confidence upper bound 77 on Eb[Jis;b,c)]. 
If 7]< Lf{s), then s can be added to the set of states, by setting 5^+1 =s and fk+i^ri. When 
desired, some 5' may be dropped when their corresponding fi values are implied by the 
values of other states, as evident by solving a problem like (PI). 



Minimize c^jc + Xfy 
x.y 

subject to Ax+Uy - s + b 
MUy > a 



(P2) 



Dx > d 
iy =1 
y >^ 
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Convergence 

Now, it can be proven that if the approximation from above cannot be improved 
by deriving more bounds from solutions of (P2), then the current approximation is exact. 
Thus, suppose for every state s: 

L{s)^E^[His;b,c)]>Lfis) 
It follows that for the states u' in the current formulation: 

;6,c)] = !(«•) 

and for any state s* , if in an optimal solution of{P2), s' = Uy, e y = I, mdy> 0, then 
also: 

i 

It follows that the function L satisfies the Belhnan equation: 

L(s) = E,^[mm{c''x-\-XL(s')}] 

x.s' 

Let X {s;b,c) and s'(s\b,c) denote optimal choices at s + b, given c, relative to the 

function L and let x(s;b,c) and s'{s;b,c) denote the optimal choices at 5 + 6 relative to 
the fiinction L. Thus: 

L{s) = E,Sc' x{s;b,c) + X I(s\s;b,c)] < x{s;b,c) + Ji I{s\s;b,c)] 

and: 

^s) = [c^x{s\ b, c) + XL{s' (5; b, c)] 
14 
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It follows that: 



0<L{s)- L{s) < E^^ [c^x{s\ b,c) + XL{s' {s\ b, c)] 
- [c''x{s\ b, c) + XL{s' (5; b, c)] 

= X{E,,[L{s\s',b,c)] - E,XL{s\s',b,c)\) 

5 Assuming the state space is bounded, the latter implies L (5) = L{s) , since otherwise a 

contradiction is obtained by choosing s so that L{s) - L(s) is sufficiently close to the 
supremum of this difference over the state space. 

Approximating the Value Function from Below 
10 The approximation from below is based on knowledge of linear frmctions that lie 

below the convex fimction L(s). Suppose v\..,y e i^'" and gi,..., gr e Ware such that 
for every s that satisfies Ms >a: 

I(5)>/,(.)-(v'f^ + g,(/ = l,...,r) 
Then, the maximum L(s) = max,- {li{s)} is a convex ftmction that bounds L(s) from 

15 below. 

An Alternate Value Function 

Because of the alternating nature of the process, i.e., alternating between chosen 
actions and random transitions, there is an alternative way to define values of states, 
20 which turns out to be helpful in the case of approximation from below. First, denote: 

Kis;c) = mm{c^x + AL(s') \ (x,s*) e F(s)} 
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so that: 

Lis) = E^[K{s^b;c)] 

It is easy to see that K{s;c) is the minimum possible expected total discounted cost, given 
c, when the starting state is s and the first change is induced by a chosen action rather 
5 than by a random transition. The Belhnan [1] optimality equation, therefore, can be 
written in terms of K in the form; 

K{s;c) = mm{c''x + AEt,^.[K{s'+b;c')] \ (x^s') e F{s)} 

Convexity 

1 0 The proof of convexity of the function K(s;c) (for a fixed c) is similar to that of 

L{s). Fix any sequence b^,b\b^,,.. of realizations of the state change vectors (except that 
b^ = 0), fix any sequence c\ <?, ...of cost vectors, and fix an optimal policy. An initial 
state now determines the future states and actions, and therefore: 

K(5^Z),c) = XA'(c''fy 

1=0 

1 5 where the action vectors satisfy Dx' >d,Ax' + s''^^ =s'^b' and Dx' > d. Another initial 
state P has: 

with Ay' + t^^^ Ui > a, and Dy' > d. It can be deduced that: 

r((l -a)^' + ar° ;Z),c) < (1 - a)K(5^^c) + aF(f° ;6,c) 
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for every 0 < a < 1, and convexity is established by taking expectations over the 
sequences of b' s and c' s. 



Bounds From Duality 

Now, the lower bounds on K{s\c) can be derived based on the Unear functions 
lis). Denote = [v^..., V] e m"''^ g = and e = (1,...,1)^ e 9i'. 

Let ^ be scalar, and denote the optimal value of the following linear programming 

problem by K{s\c)\ 

Minimize Jx + 

subject to Ax-\- s' - s (P3) 
Ms' > a 
Dx > d 
^e- Vs' > g 



Obviously, at an optimal solution, ^ has the value of 

i( 5' ) = max , {(v'y 5' + g^ <L{ s' ). It follows that K{s\c) <K{s\c\ but further 
information can be derived by considering the dual problem of (P3), i.e.: 



■Maximize s y-^ a z + d w + g p 
y,z,w,p 

subject to A^y +D^w = c 

y + M^z-'V^p^O iD3) 
e^p = X t 
z, w, p >0 
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The function K(s,c) is piecewise linear and convex since the feasible domain of (Z)3) is 
independent of s, and the maximum is always attained at a feasible basis; so, K(s ; c) is 
the maximum of linear functions of s corresponding to feasible bases of (D3). Let 
y=y{s ;c), z = z{s ;c), w = w{s ;c) and p=p(s ;c) denote the optimal solution of {D3) 
when s=s . Since this solution is feasible for every s, it follows that for every s, 
because K(s;c) is also the maximum value (D3), the following holds: 

K{s;c) > K{s\ c)>y^s + a^z + d''w + g^'p 
Replacing s by s+b, the latter can also be expressed by stating that for every s and b: 

K{s + b;c) > y^s + y^b-\-a^z + d^w-}- g^p 
Now, let v'"''' = Ec\yl and let: 

gr.i =E,J,y''b] + a''z-¥d''w+g^p 

so that: 

E,J,K{s + b',c)] = {V^'Ys^g,^, 

Since L{s) = Et[K{s + b;c)l it follows that L(s) >lr+\(s) = (V^Ys + gr+i- 

Ub and c are stochastically independent, then the expectation Ebc\y^b] can be 
easily calculated as ^ ^Eiy i]E[bi] , and each E[bi] can be estimated in advance. 

If Z; (s) is sufficiently close to L(s\ then an approximately optimal action at a state 
5 + 6 can also be calculated by solving (P3), 

Convergence 
Denote: 
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L^s) = max {li(s)} 
\<i<r 



and suppose the (r + l)th linear function is obtained while s is the next state that is 
calculated while solving (P3), i.e.: 



Obviously, as long as there exist s and s such that lr+\(s; s)>Lr then a better 
approximate value function Z^+i (s) can be obtained. Otherwise, further iterations from 
below would resuh in the same approximate value function Lr (s). 

Denote L{s) = Ebc[K(s + Z7 ; c)] and suppose that (P3) cannot improve the 
approximation firom below at any state, that is, for every s, L(s) <Lr(s), It can now be 
shown that in this case Lr (5) is indeed the correct value L{s). By definition, K(s;c) is 
obtained by solving (P3), which also determines a certain poMcy x = x(s;c) and a certain 
next state s' = s'{s ;c). Denote by 7:(s ;c) the expected total discounted cost, starting at s 
with cost vector c (just before an action has to be chosen) and using the pohcy x(.). Then, 
for all s: 



and lr+\(s ;i ) is the optimal value of (P3). Then: 



Lj-+\{s) = max { L^s) , lr+\(s) } 



n{s;c) = Jx(s) + XE^c' [ n{ 5' {s;c) + ; c') ] 



It follows that for all s: 



On the other hand, since L{s) is the optimal value, for all s: 



E,Ms;c)]>L{s)>L,(s) 
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It follows that: 



0>L,is')-E, [TTis' ; c)] > M {E[L, (5'" )] - E[7c(S' ; c)]) 



where S ' denotes the state that is reached after / steps, while implementing the policy 
x{.), so the expectation is taken over all the random parameters involved. If the state 
space is bounded, the sequence E [Z^5 ') - n(S '; c')] is also bounded and, if < 1, it 
follows that L^s^) = Ec Ms^ ;c)] = L{s% 

Learning The Value Function 

The value function can be approximated successively. Iterations of the successive 
approximation algorithm can be executed both during the preprocessing phase, while 
simulating random state transitions, and also while the MDP itself is running in real time. 
In such simulations, actions are chosen based on the currently known approximations 
from above and from below. It is important to note that, for the sake of rurming the MDP 
optimally, there is no need to know the value frmction in its entirety. It suffices to know 
only the values of states that actually occur in the process and those that could occur if 
certain actions were taken, and even so, only the relative values are important for 
choosing the best action. In other words, sometimes values do not have to be known with 
high accuracy. Furthermore, knowledge of a lower bound on the value of one state and 
an upper bound on the value of another state may suffice for choosing the optimal action. 

Suppose the system is currently in state s just before an action has to be chosen. 
An action can be chosen based on the approximation of L(s' ) from above by solving {P2) 
or based on the approximation of L(s') from below by solving (P3). Moreover, a 
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weighted average of the two approximate value function can be used by solving the 
following linear programming problem for some constant and even analyze the 
sensitivity of the solution to fi: 



Suppose an action x is calculated as an optimal solution of one of the possible linear 
programming problems. Before an action x is actually executed, the resulting state s' can 
be further evaluated by running simulations with s' as the initial state and using any of 
the approximate value functions for computing actions. An approximate value function 
determines some policy, not necessarily optimal, so the simulation results provide an 
upper bound on the value of s' . Such a bound can be compared to the one derived by 
convexity from previously known states, and the state s' may be added to the list of 
states that are used for representing the approximation from above. 

Logic of the Preferred Embodiment 

FIG. 2 is a flowchart that illustrates the general logic of a message or event-driven 
supply chain planning process 112 according to the preferred embodiment of the present 
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invention. Specifically, the logic indicates how the supply chain planning process 1 12 
derives optimal policies during its operation. 

Block 200 represents the supply chain planning process 112 accessing a vector s 
of available resources, e.g., from a database 116, from another system, from a previous 
cycle of this process 112, etc. Note that supply chain management decisions are made 
cyclically, and thus Block 200 may represent the process 112 obtaining a vector of 
leftover resources from a previous cycle. 

Block 202 represents the supply chain planning process 112 accessing a resource 
change vector b and a cost vector c, e.g., from a database 1 16, from another system, from 
a previous iteration of this process 112, etc. The vector b comprises additional resources 
that became available since the last cycle, and this vector b is added to the vector s. 

Block 204 represents the supply chain planning process 112 employing a linear 
programming (LP) solver to compute a new action vector x. Specifically, the LP solver 
uses an approximate value function L{s^ in a linear programming formulation to 
determine an action x, e.g., how much to produce from the available resources s. The 
problem is to determine the action x, not just to maximize the immediate profits by 
minimizing c^x , but to take into account the value of the vector s of available resources 
remaining in anticipation of arrival of more resources. 

As noted above, both the state space (i.e., the vector s) and the action space (i.e., 
X{s)) are continuous. Moreover, the state space and the action space are related to each 
other through a system of linear constraints, wherein one or more actions in the action 
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space that are feasible in a state of the state space are linearly constrained with respect to 
the state. 

Specifically, this Block builds one or more approximations from above and from 
below to a value function L(s) for the state s using representations that facilitate the 
computation of approximately optimal actions at any given state by linear programming. 
These approximations can be repeatedly refined in an iterative manner. The value 
function L{s) is convex, which means that it can be efficiently learned in advance and can 
be represented in a way that allows for real-time choice of actions based on it. Once the 
value function L{s) is approximated, an action jc can be selected. 

Block 206 represents the supply chain planning process 1 12 executing the actions 
described by the vector x, wherein the vector s' represents the leftover resources 
remaining after the actions described by the vector x have been performed. 

Block 208 represents the supply chain planning process 112 setting the vector s to 
the vector 5' for the next cycle. Thereafter, control transfers back to Block 200. 
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5 

Conclusion 

This concludes the description of the preferred embodiment of the invention. In 
summary, the present invention discloses a method, apparatus, and article of manufacture 
for solving stochastic control problems of linear systems in high dimensions by modeling 

10 a structured Markov Decision Process (MDP). A state space for the MDP is a 

polyhedron in a Euclidean space and one or more actions that are feasible in a state of the 
state space are linearly constrained with respect to the state. One or more approximations 
are built from above and from below to a value function for the state using 
representations that facilitate the computation of approximately optimal actions at any 

1 5 given state by linear programming. 

The foregoing description of the preferred embodiment of the invention has been 
presented for the purposes of illustration and description. It is not intended to be 
exhaustive or to limit the invention to the precise form disclosed. Many modifications 
and variations are possible in light of the above teaching. It is intended that the scope of 

20 the invention be limited not by this detailed description, but rather by the claims 
appended hereto. 
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